Smaller Architecture Search Spaces

To simplify neural architecture creation, AutoML is gaining traction from evolutionary algorithms to reinforcement learning or simple search in a constrained space of neural modules. A big issues is its computational cost: the size of the search space can easily go above ̃10 candidates for a 10-layer network and the cost of evaluating a single candidate is high even if it’s not fully trained. In this work, we use the collective wisdom within the neural networks published in online code repositories to create better reusable neural modules. Concretely, we (a) extract and publish GitGraph, a corpus of neural architectures and their descriptions; (b) we create problem-specific neural architecture search spaces, implemented as a textual search mechanism over GitGraph and (c) we propose a method of identifying unique common computational subgraphs. 1 BETTER AUTOML SEARCH SPACES Current automated neural architecture creation strategies rely on extensive expert knowledge and heavy handed supervision. They either use predefined modules Negrinho & Gordon (2017) and the novelty lies in the recombination or they create new modules but within a very tightly controled structure Zoph & Le (2016); Such et al. (2017). The reason for this heavy-handed supervision is that each step taken towards a better architecture is costly. This constraint is independent of the search method used. Whether it’s employing reinforcement learning Zoph & Le (2016); Baker et al. (2016) or evolutionary algorithms Such et al. (2017), for each change the system must evaluate candidates and each evaluation means training a full network on a usually complex task. The smaller the changes, the more candidates need to be evaluated. The space of possible options is too large to allow searching or evolving a full architecture from basic building blocks like matrix additions or multiplications. Shortcuts are thus necessary. Neural evolution can be seen as a combination of two problems defining a neural module search space and creating a policy to create that space. The question of finding the right policy has received almost all the community’s attention Negrinho & Gordon (2017); Such et al. (2017); Zoph & Le (2016), with the search space receiving almost none. Notable exceptions are Schrimpf et al. (2017), who explicitly state that different domains require different operators, that are subsequently combined to form neural architectures and Negrinho & Gordon (2017) who allow experts to state what are the modules to use for a task. We propose constructing the search space by using the known architectures for similar tasks. Expert supervision can guide the search and lower the network creation cost. In our view, however, this supervision need not be a laborious task linked to the task at hand. Instead, it can come from repositories of computation graphs previously published for similar tasks. As shown in figure 1, we split the task of search space definition into three parts: 1. Search for architectures that solve similar problems. This step yields a collection of graphs. 2. Common Subgraph Mining. Extract the neural modules and combinations of modules that are common between the found architectures, like convolution + Max pool + affine. 3. Defining the Search Space by specifying which modules are large, frequent and unique enough to be useful. These subgraphs then become a toolbox of task-oriented modules. The resulting task specific module toolbox becomes the starting point to evolve new architectures.